Evaluating Search Features: Lessons from Launching Knowledge Panels & Vector Search in Medical Search

“As professional expert site search engines evolve, they are increasingly adopting features pioneered by general-purpose platforms such as Google - knowledge panels, preview snippets, LLM-synthesized answers, and vector search. In essence: pulling up answers to questions to the search result pages. However, evaluating these features, particularly in the absence of transactional signals like purchases or basket additions, remains a significant challenge. How can we determine whether users have found what they were looking for?

This case study examines how AMBOSS, a medical information platform used by 60% of US medical students and hundreds of thousands of doctors in the US and worldwide, developed and deployed search features such as Knowledge Panels, People Also Searched, and Vector Search. I discuss the challenges of measuring success in a domain where traditional relevance signals such as basket value or purchases are not available and delve into alternative approaches to evaluation.

For example Click-through rate (CTR), often used as a proxy for search relevance, can be both an indicator for relevant search results but also for irrelevant answer snippets shown, depending on context. Through a combination of user feedback loops, and engagement-based heuristics, we developed a framework for assessing search feature effectiveness that we use in online controlled experiments (A/B testing). This talk provides practical insights into judging impact of algorithmic improvements and user experience changes in a high-stake, niche domain, knowledge-intensive search environment.”

Valentin v. Seggern

AMBOSS SE

Valentin v. Seggern is a Senior Principal Product Manager at AMBOSS, where he leads the development of AI-driven search and recommendation systems that empower medical students and professionals with precise, on-demand knowledge. With over two decades of experience spanning software engineering, product strategy, and AI applications, Valentin has built cutting-edge solutions at the intersection of technology and healthcare. Before AMBOSS, he held leadership roles at Ada Health and T-Mobile, blending deep technical expertise with a passion for user-centric innovation.